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ABSTRACT 

The potential application of objectives and 
objective -ba sed measures to evaluation problems of different vypes is 
discussed. A framework for categorizing educational evaluation 
problems and decision areas is used to support the discussion. 
Decision areas are: 1) selection of the appropriate problem or 
objective to be served; 2) selection and design of the program to be 
introduced which best fulfills the objective; 3) modifying the 
urogram in terms of field conditions: and 4) certifying the 
appropriateness cf the program for introduction elsewhere. Five 
evaluation stages are utili7ed to provide information related to 
these decisions: i) Needs Assessment Fvaluation; 2) Program Analysis 
Fvaluation; "*) Implementation Analysis Evaluation; 4) Process 
Analysis Evaluation; and F) Outcome Analysis Evaluation. Each stage 
is described and the relationships between evaluation information and 
subsequent decisions ere noted, as well as the fact that objectives 
and objective-based measures are of major importance in evaluation. 
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The main purpose of this paper io to discuss the potential application 
of objectives and objective-based measures to evaluation problems of dif- 
ferent types. To accomplish this, a framework must first be established 
for categorizing and then considering the range of educational evaluation 
problems . 

At the Center for the Study of Evaluation a definition of evaluation 
has been adopted which focuses on the necessity for meeting the informa- 
tion needs of decision-makers. That is, evaluation is the process of 
selecting, collecting, analyzing and reporting information in a meaning- 
ful form that will enable decision-makers to select among alternatives. 

This general view of evaluation has forced the Center to view the 
total instructional improvement process in order to identify the normal 
sequence of educational decisions that might occur. The identification 
of the educational decision sequence (or of one possible educational 
decision sequence) provides insight concerning the kinds of judgments 
that decision-makers Mist make end the concomitant information needs of 
these decision-makers. 

In Chart I, which follows, four major decision areas are identified 
and the five kinds of evaluation which provide information for decisions in 
these areas. * The decision areas deal with: (1) selection of the appropriate 

problem or objective to be served; (2) selection and design of the program 
to be introduced which best fulfills the objective; (3) modifying the pro- 
gram in terms of field conditions; and (4) certifying the appropriateness 

*hhile the listii.g in Chart 1 seems to imply a linearity between evalua- 
tion and decisions and between decision stages, CSE would not be so 
naive as to believe that such a relationship always pertains. The eval- 
uation-decision model has been simplified for discussion purposes here. 



Chart 1 



Decision Areas and Evaluation Requirements 



Evaluation 


Decision Area 


Needs Assessment (Ep 


Problem Selection (Dp 


Program Analysis (Ep 


Program Selection (D 2 ) 


Implementation Analysis (E, ,) 
(Intervention) 

Process Analysis (E, p 
(Intervention) 


4 

Program M<odification (Dp 


(Xitcome Analysis (H.) 
(Non-intervention) 4 


Program Certification (Dp 
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of the program for introduction elsewhere. The five evaluations listed are 
intended to provide information related to these decisions. In the succeed- 
ing paragraphs each of these areas will be xamined in turn. 

Needs Assessment Evaluation (E^) 

The decision-maker may be forced to make choices related to school or 
program priorities. This is generally a reasonable assumption. Given the 
scarcity of resources within his current situation, a decision-maker might 
want to make a decision about the problem area most in need of attention. 
This problem selection decision is based in large part on a needs assess- 
ment evaluation . Needs assessment attempts to examine the gap between 
goals and the existing state of affairs. The evaluative problem is 
essentially one of assessing the needs of students, of the community, 
and of society in relation to the current status or accomplishments of 
the system. 

Needs assessment is objective-oriented in that the assessment must 
result in a statement of desired educational objectives that are compared 
with actual educational attainments related to the same objective dimensions. 

Having been provided with this information, the decision-maker con- 
cerned with the instructional improvement process is able to select from 
among the alternatives those problem areas (objective dimensions) which 
need attention or modification. Further exemplification of the use of 
objectives and objective-based measures in needs assessment evaluation 
is to be found in CSE Peprint Series, Reprint No. 10, ’'Experimental 
Assessment of the Effects of the PROBE System," by Eva Baker. 

Program Analysis Evaluation (E^) 

A second decision area of concern deals with program selection. The 
decision-maker, having made a decision about the specific problem area (or 
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objective) in which his system is deficient, must choose between the existing 
program which is directed towards the achievement of that objective and 
various alternative programs that might be introduced. These choices are 
referred to as program selection decisions . These decisions are in largo 
part based upon information provided in a program analysis evaluation . 

In this evaluation stage, the evaluator is generally asked to provide 
information on the possible impact of the introduction of several alter- 
native programs. The evaluator's task is to provide, prior to a program's 
inception, information concerning the potential success of that program. 

Thus, while it may appear that the evaluator's job is merely speculative, 
he must attempt, at the very least, to develop procedures for systematizing 
speculation. Review procedures foi examining the quality of the program 
materials might be developed. Alternatively, simulations and games might 
be utilized to examine in microcosm the potential impact of the various 
alternatives. 

A program analysis evaluation is objective -oriented in that the alter- 
native programs, including those which are developed during the evaluation, 
are being analyzed relative to their potential accomplishment of the desired 
objective(s) . Moreover, potential attainment of other objectives (desired 
or undesired) is also a basis for consideration of programs. 

When such information has been provided, the decision-maker must make 
a judgment as to which program offers the greatest probability of success 
within his system. The decision is not completely dictated by the data 
provided by the evaluator. Even though the evaluator should attenpt., 
within his study, to be as cognizant as possible of the political and con- 
textual variables which bound or restrict the nature of the ultimate 




4 



decision, it is not likely that he cm become aware of them all. Thus, the 
program actually selected may differ somewhat from the one which looks 
best "on paper." 

Implementation Analysis (E^ and Process Analysis (E^ 2) Evaluations 

Having made the decision to introduce a specific program, the decision- 
maker must be in a position to make program modifications, as the situation de- 

I 

mands, throughout the course of the introduction of that program. These pro- 
gram modification decisions require evaluation information of various types. 

* 

First, there is a need for information concerning the extent to which the 
program has been implemented in the manner prescribed during program selec- 
tion and to the group for whom it was intended. This information is 
referred to as implementation analysis evaluation . Secondly, the decision- 
maker must have information on the impact of the program on the educational 
process. This pr ocess analysis evaluation is intended to provide infor- 
mation to the decision-maker on the progress being achieved towards ful- 
filling the intended objective(s) as well as information on any unantici- 
pated outcanes which might have been observed. 

In an implementation analysis evaluation the concern is with pro- 
viding information on the manner in which the program has been implemented. 

One relevant question is the degree to which that program, as described in 
the program selection decision, has been introduced. That is, has 
the program been introduced and is it opeiating in the pedagogical man- 
ner in which it was intended to be used? t\»rthermore, the program selec- 
tion decision was based on various assumptions about the nature of the 
student population to be served. If these assumptions were incorrect or 
are no longer appropriate, then this information, as a part of an 
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implementation analysis evaluation, should be meaningful to the decision-maker 
in determining whether it is appropriate to allow the program to continue. 
Thus, the nature of the student population is another source of data in an 
implementation analysis evaluation. From the above, it is apparent that 
implementation evaluation is not directly objectives- related, in that the 
evaluator does not consider the extent to which the objectives have been 
achieved in this evaluation stage. 

In a process analysis evaluation, the evaluator has been called upon 
to provide data on how the program is functioning. In terms of the short 
range objectives of the system, such as the intended objectives to be 
achieved at the end of a specific unit of study, what has been the per- 
formance of the student group? Are there noticeable unanticipated out- 
comes, not a part of the original objective for the program but which, 
nevertheless, ought to be noted as important information potentially 
valuable to a decision-maker in making decisions about the program? 

A process analysis evaluation is objectives -oriented. The main data 
items obtained by the evaluator are the progress towards the achievement 
of the objective(s) and unanticipated outcomes. 

The decisions related to implementation analysis evaluation and pro- 
cess analysis evaluation are likely to be program modifications during 
the course of the program rather than decisions at the conclusion of 
the program. The evaluative functions identified in the two types of 
evaluation leading to program modification decisions are designed to te 
interventionist in nature. That is, during these stages tlie evaluator 
is envisaged as one who is concerned with providing information regarding 
the modification and improvement of programs during the process of their 
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introduction. This is directly opposed to the more passive role that might 
be associated with a research-observer who, through his desire to draw 
valid generalizable conclusions, is careful not to intercede in the process. 

Outcome Analysis Evaluation (E^) 

At some point, after a program has been introduced into the system and 
has been properly implemented and modified in line with whatever difficulties 
have been noted, one may wish to consider the potential generalizability of 
the program. A decision related to the potential generalizability of a 
program to other educational systems is viewed as a program certification 
decision . The evaluation associated with such decisions is referred to as 
outcome analysis evaluations . 

In outcome analysis evaluations the role of the evaluator is modified 
from the interventionist stance previously described. In order to maintain 
the generalizability of the situation, it is imperative that the evaluator 
not be actively involved in the program, and that he attempt to ensure that 
drastic program modifications are not being made concurrently with the 
evaluation. Most protocols of experimental research would be applicable 
to such situations. It is the view of the Center that the changing nature 
of educational systems as well as the exigencies of real world problems 
would make it extremely difficult to complete a pure outcome analysis 
evaluation. 

Summary 

In the preceding pages , a framework has been presented which depicts 
an evaluation- d cision sequence. Each of the evaluation stages has been 
described, and the relationships between evaluation information and 
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subsequent decisions have been noted. In Chart 2, the evaluation information 
appropriate to each stage has been summarized. All stages, with the 
exception of implementation, utilize objectives or objective based measures 
as a prominent part of the evaluation information. In process analysis and 
outcome analysis evaluation, achievement of the objectives (use o£ objective* 
based measures) is the major source of evaluation information. In program 
analysis evaluation, data related to the likelihood of achieving the objec- 
tives are the major sources of information. Finally, in needs assessment 
evaluation the objectives themselves and the ob iective measures are the 
sources of data. 

It is apparent that objectives and objective -based measures are of 
major importance in evaluation. Recognition of the different kinds of 
decisions that evaluation serves and tbe information requirements of each 
will help in understanding the potential areas of use of objectives and 
objective-based measures in evaluation- 
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Chart 2 



Use of Objectives in Various Evaluation Stages 



Evaluation Stage 


Evaluation Information 


Needs Assessment (E^) 


Objectives, relative to outcomes, 
within constraints of system character- 
istics 


Program Analysis 


Programs, relative to likelihood of 
their achieving an objective(s) 


Implementation Analysis (E, ^) 
(Intervention) 


Implementation of Program: Personnel, 

Physical, Pedagogical 


Process Analysis (E, 
(Intervention) 


Progress toward the achievement of 
objective (s) 

Unanticipated outcomes 


Outcome Analysis (E.) 
(Non- Intervention) 


Achievement of objective (s) 
Unanticipated outcomes 
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